python-ecosys/debugpy: Add VS Code debugging support for MicroPython. - #1022

CDCInterface.__init__() sets self._timeout = 1000, then calls
self.init(**kwargs). The init() method had timeout=None as default,
which unconditionally overwrites self._timeout with None. This causes
TypeError in read(), write(), readinto(), and ioctl() which all compare
int >= self._timeout.

Set the default timeout=1000 in init() to match the intended default,
consistent with how other parameters (baudrate, bits, etc.) have their
defaults specified directly in the init() signature.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>

Copied from CPython v3.14.6 with no changes.

Signed-off-by: Damien George <damien@micropython.org>

And run it as part of CI.

Signed-off-by: Damien George <damien@micropython.org>

Using `bytes.hex()` eliminates an import, and eliminates the call to
`.decode()` to convert it to a str.

Also run ruff format.

Signed-off-by: Damien George <damien@micropython.org>

Signed-off-by: Damien George <damien@micropython.org>

Signed-off-by: Damien George <damien@micropython.org>

Use the same variable-byte Remaining Length encoding as publish for
subscribe and unsubscribe so long topics no longer overflow a single
length byte. Fixes micropython#969.

Signed-off-by: Pablo Ventura <pablogventura@gmail.com>

Factor Remaining Length VBI and ACK wait into _send_subunsub to reduce
.mpy size after the long-topic fix.

Signed-off-by: Pablo Ventura <pablogventura@gmail.com>

Factor the Variable Byte Integer encoder into a module helper and reuse it
from connect, publish, and the subscribe/unsubscribe path to cut .mpy size.

Signed-off-by: Pablo Ventura <pablogventura@gmail.com>

Use ack_n to decide whether to include the QoS byte so ruff E711 passes
without growing the compiled .mpy size.

Signed-off-by: Pablo Ventura <pablogventura@gmail.com>

Signed-off-by: Pablo Ventura <pablogventura@gmail.com>

recv_message() stripped the header from the receive buffer as soon as the
CRLF/CRLF terminator was found, but only persisted buffer state on some
partial-read paths. When a message body arrived in a later read than its
header, the parsed-header state was lost and framing desynchronised for
the rest of the connection. Keep the header and body together in the
buffer until the whole message (header + Content-Length bytes) is
present, then slice it off. Treat an empty recv as a peer close and
EAGAIN/EWOULDBLOCK as "try later".

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>

- wait_for_client() blocks until the DAP client sends configurationDone,
  draining the socket so breakpoints set beforehand are honoured; it
  replaces a fixed sleep. Bounded timeout, logged rather than silent.
- A runtime capability probe (settrace / save_names / set_local / f_back)
  derived by exercising the interpreter, never inferred from a build or
  variant name; exposed via get_capabilities().
- Local variables are marked read-only (DAP presentationHint) when the
  firmware lacks frame._set_local, so clients do not offer an edit that
  cannot work; globals stay editable.
- listen() resolves the actually-bound port and never advertises port 0.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>

DAP `evaluate` requests carry a `context` field (`watch`, `hover`, `repl`,
`clipboard`, ...) that `_handle_evaluate` read but discarded, so every
request went through `eval()` only; a statement such as `x = 5` or
`def f(): ...` typed into the Debug Console failed with a syntax error
instead of running.

`evaluate_expression` now dispatches on `context`: `watch`/`hover` (and
any other or absent context) keep the original eval-only, read-only
contract unchanged. `repl`/`clipboard` try `eval()` first, so a plain
expression like `1 + 1` still returns a value, and only fall back to
`exec(expression, globals_dict)` when `eval()` raises `SyntaxError`.

The exec namespace is globals-only, on purpose: `exec(code, g, l)` binds
a top-level assignment into `l`, and here `l` is a throwaway copy of the
paused frame's `f_locals` snapshot handed back to the caller and then
discarded, so the assignment would silently vanish instead of taking
effect. Passing only `globals_dict` makes a statement's assignments land
in the running module namespace, where they are visible to the target
program after `continue`.

That globals-only exec creates a shadowing hazard: assigning a name that
is also a LOCAL of the paused frame changes the global but leaves the
local exactly as it was, which looks like a no-op from the Debug
Console's perspective. `_shadowed_local_warning` detects the common case
(a simple `name = ...` or `name op= ...` at the start of the statement)
and appends a warning to the result so the mismatch is visible rather
than silently misleading; it does not attempt to parse multi-target
assignment, unpacking, attribute/subscript targets, or `def`/`class`/`for`
bindings, and a `None` result from `_assigned_name` means "not proven
safe", never "proven no shadowing".

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>

listen() bound the socket, blocked in accept() and handled the client's
initialize request before returning, so a caller could only learn the
endpoint after a client had already connected to it - unusable for any
orchestration that has to read the address in order to attach.

listen() now returns as soon as the socket is bound. The accept and the
initialize handshake move into wait_for_client(), which creates the
session. This matches CPython debugpy, where listen() reports the
endpoint and wait_for_client() blocks.

port=0 now raises instead of substituting DEFAULT_PORT when the target's
getsockname() cannot report the assigned port: callers act on the
returned endpoint, so naming an address the socket is not bound to sends
them somewhere nothing is listening.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>

process_pending_messages() set a 1 ms socket timeout and restored blocking
mode in its finally. The trace function calls it on entry to every new
frame, so handling a message re-enters it, and the inner call's finally put
the socket back into blocking mode underneath the outer loop. That loop's
next recv() then waited for a message the client will not send until it has
seen an event the loop itself is what produces - a deadlock between the two
sides.

It only bites when the clobber lands inside the window after
configurationDone, which is why it presented as a load-sensitive flake: the
session hangs before wait_for_client() returns, so the target never runs
and no stopped event is ever produced.

The nesting is tracked rather than the timeout saved and restored, because
MicroPython sockets have no gettimeout().

Measured on the wrapper repo's harness: the previously worst-affected file
went from 4 clean runs in 6 to 6 in 6, and the full suite from 0 clean in 3
to 3 in 4.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>

`ruff format --diff .` - the second command in this repository's ruff
CI job - reports this file as unformatted, so the job is red on this
branch.

The over-length line carries two trailing comments. Parenthesising the
value, which is what the formatter does, is the worse of the two
readings; the prose comment moves above the assignment instead, and the
`# type: ignore` stays on the line it applies to. `tools/codeformat.py`
then leaves the tree unchanged.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>

`ruff check .` reports EXE001 on all three, and the rule is pointing at
a real inconsistency rather than a style preference.

`demo.py` and `test_vscode.py` are run as `micropython demo.py`, so
`#!/usr/bin/env python3` names an interpreter that cannot run them; the
line is removed. `dap_monitor.py` is a host-side CPython tool whose
shebang is correct, so it becomes executable instead.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>

The repository's codespell job fails on this branch, and all four hits
are in this file's comments: "debugee" twice, "inialization", "Nothe".

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>

`ruff check .` reports four T100 (flake8-debugger) failures in this
package, all of them in the two sample scripts: an `import debugpy`
apiece, and `test_vscode.py`'s calls to `listen`, `breakpoint` and
`wait_for_client`. The rule is right about these files. A program that
starts its own debug server is not the case this package exists for:
on a device the client is at the other end of a link, and something
has to bind the socket and wait for it before the program runs.

So `test_vscode.py` is now just a program to debug - no debugpy import,
no manual breakpoint, a `main()` for a launcher to call after
`listen()` and `wait_for_client()` - and the README and the development
guide give the command that does that. Two incidental repairs come with
it: the loop imported `machine`, which does not exist on the unix port
the guide tells you to run this on, and the README named a test file
that is not in the package and a build directory that its own build
command does not produce.

`demo.py`'s `import debugpy` was dead: it reaches the package through
`from debugpy.server ...` imports, so the bare import just goes.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>

`caps["serial_dap"]` reports whether this session's DAP channel is a
stream rather than a TCP socket, which is knowable exactly and is what a
host deciding whether to bridge actually needs. Probing the board for a
spare CDC instead could only ever be answered after the device had
already chosen its channel, and could disagree with that choice.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>

Local names belong to the code object that declares them, so reading
this module's own frame reports whichever compiler produced debugpy.
mpy-cross does not persist names into .mpy (LOCALNAMES_PERSIST is off
because it corrupts line numbers), so an installed .mpy copy reported
save_names False on firmware that supports it, and the launcher's
capability cross-check then rejected a correct manifest.

Compile a throwaway function at probe time and read that frame instead,
falling back to the caller's frame where there is no exec to compile
with.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>

`JsonMessageChannel` polls a socket that always carries a timeout, so an
idle poll always ends in an error rather than in data. Which error is a
property of the network stack: the unix port raises EAGAIN, lwIP - every
WiFi board - raises ETIMEDOUT once the timeout elapses. Only EAGAIN and
EWOULDBLOCK were accepted, so on a board the channel closed on its first
idle poll, which arrives immediately after `initialize` drops the
timeout to 1 ms. Measured on a PYBD-SF6W over WiFi: `recv` with no data
raises 110 at any non-zero timeout and 11 only at timeout 0.

`send_message` also called `sock.send()` once and ignored the returned
count. A socket that accepts a prefix truncates the frame
mid-`Content-Length`, which has no protocol-level symptom - the client
waits forever for a message that will never complete and misreads every
later frame as its body. Writes are now driven to completion with a
stall budget, bounded because the write runs inside the trace callback.
`StreamTransport.send` returns a byte count to match the socket contract
it presents.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>

`set_breakpoints` keeps each file's set under two keys: the path the
client sent, and the name the debuggee reports for the same file,
because `should_stop` matches on `frame.f_code.co_filename`. The client
key was replaced unconditionally, the debuggee key only from inside the
per-breakpoint loop.

DAP has no remove-breakpoint request - a `setBreakpoints` carrying an
empty list is how a client removes them - so a removal never entered the
loop and left the debuggee key armed, and the program kept stopping at
breakpoints the client had cleared. With several breakpoints in one file
the key was also reset on each pass, leaving only the last of them.

The two keys are the same string unless a path mapping is configured, so
this is invisible until a launch config sets `pathMappings`.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>

`send` writes once and returns how many bytes went out, raising EAGAIN
only when nothing could be written. `messaging.py`'s `_send_all` already
drives a whole frame out from that count.

A whole-buffer contract cannot express a partial write: it loops
internally and raises OSError 110 when the timeout expires mid-frame,
discarding the count of what has already gone out, so `_send_all`
retries from offset 0 and resends that prefix and the peer's
Content-Length framing can never resynchronise. On stm32 that hit every
DAP message over 1024 bytes: `USB_VCP.write` takes at most
`MICROPY_HW_USB_CDC_TX_DATA_SIZE` per call and reports the short count,
and `public_api.py` drops the channel timeout to 1 ms once `initialize`
is answered.

Measured on a PYBD-SF6W: a 16 KB `evaluate` response over the second CDC
in 0.20 s, 81.7 kB/s.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>

`_filename_as_debugee` and `_filename_as_debugger` are now exact
inverses of each other: one mapping list, first match wins in both
directions, and a mapping matches a path only at a path-separator
boundary. A bare string prefix let a sibling directory sharing the
root's name (`/home/dev/src-old` under root `/home/dev/src`) be
rewritten into a device path that cannot exist, and
`_filename_as_debugee` kept rewriting after its first match while its
inverse stopped at one, so a second mapping could be applied to a path
the first had already translated.

`_filename_as_debugger`'s two loops were identical, making the second
unreachable; one remains.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>

`process_pending_messages` restored the channel's blocking mode in a
`finally`, and the loop it guards is what closes that channel: a DAP
`disconnect` request handled there runs the whole session teardown, so
on the way out of that request the socket is gone and `settimeout`
raises EBADF. Every caller is `_trace_function`, so that exception
landed in whichever line of the debugged program was being traced and
killed it with a traceback naming an errno belonging to the debug
channel. Deterministic for every session that ends the ordinary way, and
downstream it made mpremote's mount teardown read the dead program's
traceback as a device fault.

The restore now happens only against a channel that is still open, and a
channel found closed ends the session rather than leaving a trace
function installed to pump it.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>

MicroPython's return event reports the last line the frame executed,
where CPython's reports the line that ended the frame. should_stop
compared the line against the breakpoint table for every event, so a
breakpoint on a function's final line stopped twice per call: once as
the line event about to run it, once as the return event on the way out,
on a frame that has already produced its value and where stepping or
inspecting locals means nothing.

A hit is now recognised for call and line events only. call has to stay:
it reports the def line, so it is the only event that can ever match a
breakpoint placed there.

The dead first half of the check goes with it. It re-read the same two
frame attributes, tested `self.paused` and the breakpoint table, and
then only printed - reaching no return of its own and falling through to
the real check below. `paused` is written in three places and read in no
stop decision anywhere, so the DAP pause request answers success and
never stops the target; that is a separate gap, not something this line
was providing.

Verified against a real event stream under a built unix firmware by the
mpy-debugpy host harness, which asserts that a return event was
delivered at the breakpoint line, so the test cannot pass vacuously on a
firmware that never produced one.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>

A client's restart button now re-runs the program in the same session
instead of being refused. The session outlives the restart deliberately:
breakpoints live in the adapter, and a client that sends restart rather
than reconnecting does not re-send them, so keeping one session alive is
what makes them still bind on the next run, at no re-attach cost.

restart is offered only when the code that owns the run loop has said it
can re-run its target (enable_restart(), before wait_for_client(), since
the capability is answered during initialize). Without that,
supportsRestartRequest is false and a restart request is refused with a
message saying why - a restart button that silently does nothing is
worse than one that is absent.

The unwind is a RestartRequest raised from the trace function, because a
restart cannot wait for the target to return: the ordinary embedded
shape is a main loop that never does, and with no second thread and a
message pump that runs inside the trace function, a deliberate raise is
the only mechanism there is. It derives from BaseException so a target's
own `except Exception:` cannot swallow it and leave the restart silently
undone. The handler releases a target parked at a breakpoint and clears
any pending step, so it does not stop again on its way out, and sends
`continued` so the client's UI does not stay stopped on a frame that is
about to cease to exist. wait_for_restart() pumps messages between runs,
nothing else being able to read the socket then, and reports the client
leaving as False, so a session with nobody left to restart for ends
rather than spinning.

console() sends text to the client's debug console as a DAP output
event. It is the only route a target's own notes have to the user on a
transport where device stdout never reaches the host: a mounted serial
session's filesystem pump discards everything the device prints.
`terminated` is still never sent between runs - a client that sees it
tears the session down, which is the opposite of what a re-runnable
session is for.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>

The feature list advertised "Pause/continue execution" and "locals
generally not supported". Neither holds: `pause` sets a flag that no
stop decision reads, so a running target is never stopped by it, and
locals are readable whenever the firmware reports `save_names` (under
real names) or not (as positional placeholders). The requirements
section now also names the macro that decides which of those two a build
gives.

`_is_placeholder_local_name`'s docstring described the placeholder
format as `local_1`, `local_2` and attributed it to
`MICROPY_PY_SYS_SETTRACE_SAVE_NAMES`. The format is `local_%02d`,
0-based (`py/profile.c:183,207`), and the macro on this lineage is
`MICROPY_PY_SYS_SETTRACE_LOCALNAMES`. The check itself tests the digits
rather than the width, so it was already correct for either spelling;
that is now stated instead of implied.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>

The guide is what someone reads to find out how to watch the DAP
conversation. Three of its claims did not hold, each checked against a
live session rather than by reading:

- The server's `[DAP] RECV`/`SEND` trace was presented as unconditional.
  It is gated on `debug_logging`, which is set from the `attach`
  request's `logToFile`, so without that flag only a handful of
  unconditional `[DAP]` progress lines appear - and because the flag
  arrives with `attach`, nothing up to and including that request is
  ever traced. The first logged line of a session is the attach
  response, and the example now shows what a run actually printed.
- The expected sequence put `configurationDone` before `attach`. No
  order is enforced, but that one misleads about what gates the run:
  `configurationDone` is what releases the debuggee, which is why
  breakpoints sent before it are in place when the program starts.
- `attach` was implied to be what makes tracing happen. It carries
  `pathMappings`; `debug_this_thread()` installs the same trace
  function, so a client that never attaches still stops at breakpoints,
  it just gets no path translation. Confirmed by removing the attach
  request from a session: it still stopped.

The guide also led with the standalone `dap_monitor.py`, which needs a
second port and a re-pointed client. `mpremote debug --dap-log` records
every frame for every transport with neither, so it leads now; the
server's own logging follows as the method that needs nothing but the
firmware, and the monitor stays as the way to drive a session without
mpremote. The host-specific paths are gone.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>

`listen_stream` takes an optional `is_connected` callable, and a stream
transport that has one treats it going false as EOF, so a session whose
client disappears ends instead of waiting for a message that cannot
arrive.

A USB CDC interface never reaches EOF: an idle one and one whose host
has vanished both read as no bytes. A target stopped at a breakpoint
therefore sat in `wait_for_continue` until the board was power-cycled,
even though that function already handles a closed channel. On stm32 the
signal the interface has instead is `USB_VCP.isconnected()`, its DTR
line, which the host raises when it opens the port and the kernel drops
when the last opener goes away.

The callable is the caller's because it is port-specific, and it counts
only once the channel has carried a byte: nobody holds the interface
between `listen_stream()` and the client's first connect, and a host may
open it briefly beforehand just to check that it can, so down on its own
says nothing. A stream with a real EOF passes nothing and is unchanged.

Signed-off-by: Andrew Leech <andrew@alelec.net>

`should_stop` consumes a pending pause at the next `line` event and
reports the stop as `pause`, so a client that presses pause gets a target
that has actually stopped.

It did not before. `_handle_pause` set two `paused` flags and answered
success, and no stop decision read either, so the client's UI went to the
stopped state while the program ran on - and then asked for `stackTrace`,
`scopes` and `variables`, all answered from whatever frame the last trace
event had left in `current_frame`. The user was shown a frame the program
had already left.

Only `line` events: `call` reports the `def` line before the body has
run and `return` reports a frame that has already produced its value, the
same distinction the breakpoint check makes. A pending step is dropped
with the pause, since a step that outlived a user interrupt would fire
later at a point nobody asked for. `wait_for_continue` clears the flag on
the way out, so a pause that arrives while the target is already stopped
does not stop it again one line into its next run.

The stop reason now comes from the adapter, which is what made the
decision - a stop is a breakpoint, a consumed pause, or a landed step,
and nothing else. That makes `DebugSession.paused` and
`DebugSession.stepping` removable rather than write-only: between them
they were assigned twelve times and read once, which is what let a
request that did nothing read as implemented.

A pause still only lands where Python is running. A target blocked in
`time.sleep`, waiting on a socket, or inside a long-running C function
produces no trace event, so the request stays pending until one comes.
The README says so in place of the entry that said pause does nothing.

Signed-off-by: Andrew Leech <andrew@alelec.net>

A board with one UART and no network has one way in, and it is already
carrying the REPL. `ReplMux` splits that stream into two façades: the
program's console, and the byte stream `debugpy.listen_stream()` wants.
Whichever object the runtime diverts stdout into gets the console side;
the DAP side is handed to `listen_stream` unchanged, so nothing above
this file knows the stream is shared.

The wire reuses `mpremote mount`'s marker and code namespace so one demux
point can eventually serve both: `0x18` keeps its meaning, codes 1..13
stay the filesystem RPC's, and a DAP frame is code 14 with an explicit
two-byte length. Carrying the length rather than reading the DAP
message's own `Content-Length` keeps the reader from ever inspecting a
payload, so the only content assumption on the wire is about `0x18`, and
doubling that byte in console output removes it.

Inbound bytes are credited back to the sender. The receive path this
rides on is a fixed ring that discards the tail of a packet it has no
room for instead of exerting back-pressure, and a target inside
`time.sleep()` drains nothing, so a sender that ignored the credit would
lose the middle of a `setBreakpoints`. Blocking instead is recoverable.

`_emit` never raises. The console façade sits in the runtime's stdout
diversion, and on the ports this reaches an exception out of a diverted
`write()` deactivates the diversion - which on a single-stream board
removes the console and the debug channel in the same instant. A short
write leaves the rest queued.

Both façades are `io.IOBase` and answer `MP_STREAM_GET_FILENO` with
`-EINVAL`. Answering with a number would let a port built with
`MICROPY_PY_SELECT_POSIX_OPTIMISATIONS` poll that descriptor directly and
never call the façade's `ioctl` again, which is the only place the demux
is pumped.

Signed-off-by: Andrew Leech <andrew@alelec.net>

`StreamTransport`'s host-has-gone signal could not end a wait that had
not already carried traffic, and a `recv` given no timeout never re-read
it. On a dedicated DAP interface that leaves a session nobody was using;
on a stream shared with the REPL it takes away the console the board is
reached by, because the framing wrapper stays in the runtime's `dupterm`
slot until the session ends. Measured on a PYBD_SF6: a session killed
before any client sent `initialize` left the board answering every later
REPL in framed bytes until it was power-cycled.

The signal now also arms when the channel was already held at the moment
the transport was built. That is what separates the two cases. Nothing
holds a dedicated interface between `listen_stream()` and the client's
first connect, so its line being down says nothing until a byte has
crossed; a shared stream's hold predates the channel, so only the host
leaving can drop it, and waiting for traffic there means a client that
never attaches holds the stream forever.

A wait asked to block forever is served in slices, because the line is
not something a poll can wait on and one unbounded poll never looks at
it again.

`_accept_and_initialize` retries a partial `initialize` rather than
reading it as a client that sent some other command, and tears the
session down when the channel closes rather than building one on a
message that never arrived.

Signed-off-by: Andrew Leech <andrew@alelec.net>

The key reported whether a session's DAP channel was a stream rather
than a TCP socket, which existed to tell a dedicated serial interface
apart from the network. That transport is gone; the one remaining
stream channel reports itself as `repl_dap`, from the boot script that
knows it split the REPL.

`probe_capabilities()` therefore takes no argument and answers the same
whenever it is called: every key in it is now a property of the
firmware rather than of the run.

Signed-off-by: Andrew Leech <andrew@alelec.net>

Every value describes the firmware, which cannot change while it is
running, but the probe ran on each call - once for the boot script's
handshake and again when a client attaches - and it is not free:
`_probe_local_names` compiles source on the device to measure the
firmware's own compiler rather than whatever produced this module.

Memoising was wrong while the function took a per-session argument. It
no longer does. A copy is handed out so a caller adding its own key -
the boot script adds `repl_dap` - cannot reach the cache.

Signed-off-by: Andrew Leech <andrew@alelec.net>